Improving Vignette Descriptions and Question Formats to Measure Distance Vision: Evidence from Cognitive Interviews among Students in China

Vignette design has been largely neglected in anchoring vignette studies. This study aimed to contribute to the science of vignette design by developing and evaluating vignettes for measuring vision in rural China. Cognitive interviews were conducted among 36 participants in a Chinese middle school. The respondents either directly evaluated vision of the vignette character (i.e., noncomparative judgment) or compared their own vision with that of the vignette character (i.e., comparative judgment). It was found that a hypothetical person in the vignette was successfully envisioned by 1 Freeman Spogli Institute for International Studies, Stanford University, Stanford, CA, USA 2 National Cancer Institute, National Institutes of Health, Bethesda, MD, USA 3 Harvard T. H. Chan School of Public Health, Boston, MA, USA Corresponding Author: Yanfang Su, Freeman Spogli Institute for International Studies, Stanford University, Encina Hall E401, Stanford, CA 94305, USA. Email: [email protected] Field Methods 1-19 a The Author(s) 2017 Reprints and permission: sagepub.com/journalsPermissions.nav DOI: 10.1177/1525822X16680810 journals.sagepub.com/home/fmx participants in grade 7 and beyond. However, more than half the participants were unable to accurately estimate distances expressed in meters. Some participants were critical in self-evaluation, yet tolerant of others’ performance. Participants more easily produced an answer and had greater confidence in the answer in comparative judgment than for noncomparative judgment. We conclude with recommendations for designing concise and complete vignettes and suggest the use of comparative rather than noncomparative judgment. Measuring subjective judgments related to health, through the use of selfreport surveys, is challenging, especially because responses across respondents may be difficult to compare. To address this challenge, the anchoring vignette approach has been developed to improve interpersonal comparability of survey responses, especially to questions that use ordered categorical response scales (King et al. 2004; Salomon et al. 2001). The anchoring vignette approach relies on short descriptions of hypothetical persons with different levels on a dimension of interest to gain insights into the ways that respondents reply to particular survey questions related to that dimension. It is hypothesized that by using responses to vignettes to identify systematic differences in the ways that respondents interpret and use the categories on a given response scale, researchers can recalibrate the responses to self-assessment questions (King et al. 2004; Salomon et al. 2001). Anchoring vignettes have been advanced with tailored statistical methods (Wand 2013; Wand et al. 2011) and are used widely in the World Health Survey and in other empirical studies (Chevalier and Fielding 2011; Damacena et al. 2005; King et al. 2012; Salomon et al. 2004; Wada et al. 2011). However, vignette design, including designing vignette descriptions and question formats, has been largely neglected (Hopkins and King 2010; Kapteyn et al. 2011). More recent studies have revealed substantial violations of measurement assumptions underlying anchoring vignette techniques (Bago d’Uva et al. 2011; Grol-Prokopczyk et al. 2015; Hirve et al. 2013; Kapteyn et al. 2011), and it has been suggested that measurement assumptions are more likely to hold true if the description of the vignette character’s condition is complete and concise (Kapteyn et al. 2011). It has also been suggested that there is need for further work on improving vignette descriptions and question formats before vignettes are used in practice (Hopkins and King 2010; Kapteyn et al. 2011). 2 Field Methods